The inauguration speeches during the Great Depression and the 2007-2008 financial crisis

The Great Depression (1929-39) was the deepest and longest-lasting economic downturn in the history of the Western industrialized world. In the United States, the Great Depression began soon after the stock market crash of October 1929, which sent Wall Street into a panic and wiped out millions of investors. Herbert Hoover was the president at that time. Over the next several years, consumer spending and investment dropped, causing steep declines in industrial output and rising levels of unemployment as failing companies laid off workers. By 1933, when the Great Depression reached its nadir, some 13 to 15 million Americans were unemployed and nearly half of the country’s banks had failed. Though the relief and reform measures put into place by President Franklin D. Roosevelt helped lessen the worst effects of the Great Depression in the 1930s, the economy would not fully turn around until after 1939, when World War II kicked American industry into high gear. The financial crisis of 2007–2008, also known as the global financial crisis and the 2008 financial crisis, is considered by many economists to have been the worst financial crisis since the Great Depression of the 1930s. It began in 2007 with a crisis in the subprime mortgage market in the USA, and developed into a full-blown international banking crisis with the collapse of the investment bank Lehman Brothers on September 15, 2008. Excessive risk taking by banks such as Lehman Brothers helped to magnify the financial impact globally. Massive bail-outs of financial institutions and other palliative monetary and fiscal policies were employed to prevent a possible collapse of the world’s financial system. The crisis was nonetheless followed by a global economic downturn, the Great Recession. The Eurozone crisis, a crisis in the banking system of the European countries using the euro, followed later. Some people think George W. Bush fixed the financial crisis while some think he is one of the people to blame for. During the presidency of Barack Obama, the US revived and cleaned up its banking system. Europe, however, is still struggling to stabilize and regulate its banking system.

In this project we try to use text mining to analyze the inaugurations of these four presidents, to feel the different emotions in the speech and try to find the reflect of the time in the inaugurations.

Step 0: check and install needed packages. Load the libraries and functions.

setwd("/Users/wangzhishan/Desktop/Spr2017-Proj1-iriswang0501/doc")
getwd()

[1] "/Users/wangzhishan/Desktop/Spr2017-Proj1-iriswang0501/doc"

packages.used=c("rvest", "tibble", "qdap", 
                "sentimentr", "gplots", "dplyr",
                "tm", "syuzhet", "factoextra", 
                "beeswarm", "scales", "RColorBrewer",
                "RANN", "tm", "topicmodels")
# check packages that need to be installed.
packages.needed=setdiff(packages.used, 
                        intersect(installed.packages()[,1], 
                                  packages.used))
# install additional packages
if(length(packages.needed)>0){
  install.packages(packages.needed, dependencies = TRUE)
}
# load packages
library("rvest")
library("tibble")
library("qdap")
library("sentimentr")
library("gplots")
library("dplyr")
library("tm")
library("syuzhet")
library("factoextra")
library("beeswarm")
library("scales")
library("RColorBrewer")
library("RANN")
library("tm")
library("topicmodels")
library("dplyr")
library("tidytext")
library("SnowballC")
library("wordcloud")
source("../lib/plotstacked.R")
source("../lib/speechFuncs.R")

This notebook was prepared with the following environmental settings.

print(R.version)

               _                           
platform       x86_64-apple-darwin13.4.0   
arch           x86_64                      
os             darwin13.4.0                
system         x86_64, darwin13.4.0        
status                                     
major          3                           
minor          3.1                         
year           2016                        
month          06                          
day            21                          
svn rev        70800                       
language       R                           
version.string R version 3.3.1 (2016-06-21)
nickname       Bug in Your Hair            

Step 1: Data harvest: scrap speech URLs from http://www.presidency.ucsb.edu/.

We used Selectorgadget to choose the links we would like to scrap. We selected all inaugural addresses of past presidents, and we only use the inaugurations of four presidents here.

### Inauguaral speeches
main.page <- read_html(x = "http://www.presidency.ucsb.edu/inaugurals.php")
# Get link URLs
# f.speechlinks is a function for extracting links from the list of speeches. 
inaug=f.speechlinks(main.page)
#head(inaug)
as.Date(inaug[,1], format="%B %e, %Y")

 [1] "1789-04-30" "1793-03-04" "1797-03-04" "1801-03-04" "1805-03-04" "1809-03-04" "1813-03-04"
 [8] "1817-03-04" "1821-03-04" "1825-03-04" "1829-03-04" "1833-03-04" "1837-03-04" "1841-03-04"
[15] "1845-03-04" "1849-03-05" "1853-03-04" "1857-03-04" "1861-03-04" "1865-03-04" "1869-03-04"
[22] "1873-03-04" "1877-03-05" "1881-03-04" "1885-03-04" "1889-03-04" "1893-03-04" "1897-03-04"
[29] "1901-03-04" "1905-03-04" "1909-03-04" "1913-03-04" "1917-03-04" "1921-03-04" "1925-03-04"
[36] "1929-03-04" "1933-03-04" "1937-01-20" "1941-01-20" "1945-01-20" "1949-01-20" "1953-01-20"
[43] "1957-01-21" "1961-01-20" "1965-01-20" "1969-01-20" "1973-01-20" "1977-01-20" "1981-01-20"
[50] "1985-01-21" "1989-01-20" "1993-01-20" "1997-01-20" "2001-01-20" "2005-01-20" "2009-01-20"
[57] "2013-01-21" "2017-01-20" NA          

inaug=inaug[-nrow(inaug),] # remove the last line, irrelevant due to error.

Step 2: Using speech metadata posted on http://www.presidency.ucsb.edu/, we prepared CSV data sets for the speeches we will scrap.

inaug.list=read.csv("/Users/wangzhishan/Desktop/Spr2017-Proj1-iriswang0501/data/inauglist.csv", stringsAsFactors = FALSE)

We assemble all scrapped speeches into one list. Note here that we don’t have the full text yet, only the links to full text transcripts.

Step 3: scrap the texts of speeches from the speech URLs.

speech.list=rbind(inaug.list, nomin.list, farewell.list)
speech.list$type=c(rep("inaug", nrow(inaug.list)),
                   rep("nomin", nrow(nomin.list)),
                   rep("farewell", nrow(farewell.list)))
speech.url=rbind(inaug, nomin, farewell)
speech.list=cbind(speech.list, speech.url)

Based on the list of speeches, we scrap the main text part of the transcript’s html page. For simple html pages of this kind, Selectorgadget is very convenient for identifying the html node that rvest can use to scrap its content. For reproducibility, we also save our scrapped speeches into our local folder as individual speech files.

# Loop over each row in speech.list
speech.list$fulltext=NA
for(i in seq(nrow(speech.list))) {
  text <- read_html(speech.list$urls[i]) %>% # load the page
    html_nodes(".displaytext") %>% # isloate the text
    html_text() # get the text
  speech.list$fulltext[i]=text
  # Create the file name
  filename <- paste0("/Users/wangzhishan/Desktop/Spr2017-Proj1-iriswang0501/data/fulltext/", 
                     speech.list$type[i],
                     speech.list$File[i], "-", 
                     speech.list$Term[i], ".txt")
  sink(file = filename) %>% # open file to write 
  cat(text)  # write the file
  sink() # close the file
}

Step 4: data Processing — generate list of sentences

We try to generate list od sentences for the following analysis. For each extracted sentence, we apply sentiment analysis using [NRC sentiment lexion]. “The NRC Emotion Lexicon is a list of English words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive). The annotations were manually done by crowdsourcing.”

sentence.list=NULL
for(i in 1:nrow(speech.list)){
  sentences=sent_detect(speech.list$fulltext[i],
                        endmarks = c("?", ".", "!", "|",";"))
  if(length(sentences)>0){
    emotions=get_nrc_sentiment(sentences)
    word.count=word_count(sentences)
    # colnames(emotions)=paste0("emo.", colnames(emotions))
    # in case the word counts are zeros?
    emotions=diag(1/(word.count+0.01))%*%as.matrix(emotions)
    sentence.list=rbind(sentence.list, 
                        cbind(speech.list[i,-ncol(speech.list)],
                              sentences=as.character(sentences), 
                              word.count,
                              emotions,
                              sent.id=1:length(sentences)
                              )
    )
  }
}

Some non-sentences exist in raw data due to erroneous extra end-of sentence marks.

sentence.list=
  sentence.list%>%
  filter(!is.na(word.count)) 

Step 5: Data analysis — length of sentences

For simpler visualization, we chose a subset of four presidents on which to focus our analysis.

sel.comparison=c( "HerbertHoover","FranklinDRoosevelt","GeorgeWBush", "BarackObama")

sentence.list.sel=sentence.list%>%filter(type=="inaug", File%in%sel.comparison, Term==1)
sentence.list.sel$File=factor(sentence.list.sel$File)
sentence.list.sel$FileOrdered=reorder(sentence.list.sel$File, 
                                  sentence.list.sel$word.count, 
                                  mean, 
                                  order=T)
par(mar=c(4, 11, 2, 2))
beeswarm(word.count~FileOrdered, 
         data=sentence.list.sel,
         horizontal = TRUE,
         pch=16, col=alpha(brewer.pal(9, "Set1"), 0.6), 
         cex=0.55, cex.axis=0.8, cex.lab=0.8,
         spacing=5/nlevels(sentence.list.sel$FileOrdered),
         las=2, ylab="", xlab="Number of words in a sentence.",
         main="Inaugural Speeches Term 1") 
sentence.list.sel=sentence.list%>%filter(type=="inaug", File%in%sel.comparison, Term==2)
sentence.list.sel$File=factor(sentence.list.sel$File)
sentence.list.sel$FileOrdered=reorder(sentence.list.sel$File, 
                                  sentence.list.sel$word.count, 
                                  mean, 
                                  order=T)
par(mar=c(4, 11, 2, 2))

beeswarm(word.count~FileOrdered, 
         data=sentence.list.sel,
         horizontal = TRUE,
         pch=16, col=alpha(brewer.pal(9, "Set1"), 0.6), 
         cex=0.55, cex.axis=0.8, cex.lab=0.8,
         spacing=5/nlevels(sentence.list.sel$FileOrdered),
         las=2, ylab="", xlab="Number of words in a sentence.",
         main="Inaugural Speeches Term 2")

We assume that people in the same time tend to have more similiar speech habits. Then in Term 1 George W. Bush has more relatively short sentences than Barack Obama while Herbert Hoover and Franklin D. Roosevelt have relatively same length setences. But in the Term 2, we can see that Bush also have long sentences in his inauguration speech. That may be because 9.11 and he condemned the terrorism and build confidence of the nation to survive in the financial crisis. Sentences conveying these emotions are usually longer.

Short sentences in inaugural speeches.

sentence.list%>%
  filter(File=="HerbertHoover", 
         type=="inaug", 
         word.count<=3)%>%
  select(sentences)

sentence.list%>%
  filter(File=="FranklinDRoosevelt", 
         type=="inaug", 
         word.count<=3)%>%
  select(sentences)

sentence.list%>%
  filter(File=="GeorgeWBush", 
         type=="inaug", 
         word.count<=3)%>%
  select(sentences)

sentence.list%>%
  filter(File=="BarackObama", 
         type=="inaug", 
         word.count<=3)%>%
  select(sentences)

From the short sentences, we can clearly find the different emphasizes in each time. In Hoover’s time, the country is not as safe as it is now. Safety is the priority people care about so the president mentioned that “Crime is increasing.” to draw the attention of the people. While in Roosevelt’s time, fighting the Great Depression was the most urgent thing, so Roosevelt stated a lot about what kinds of economy policies he would use to save the country. It is no wonder that he mentioned about tax in the speech. Obama’s short sentences also tells a lot. Now US people live in a no war time and the influenc of the financial crisis is fading. After achieving the basic demand such as safety and food, people have higher level demand such as diversity and openness. From the big point, we are in a time that technolies blow out. Diversity and openness are key elements to attract the talent from all over the world. Obama was fully aware of that because he knew the value of talents. Also from the situation of US, in the preseiency of Obama, the government was respectful and supportive of L.G.B.T.Q. rights, which is a big step of the shole society.

Step 5: Data analysis — sentiment analysis

We try to get a direct impression of the cluster of emotions using the following graphs. ## Clustering of emotions

sel.comparison=c( "HerbertHoover","FranklinDRoosevelt","GeorgeWBush", "BarackObama")
my_palette <- colorRampPalette(c("red", "yellow", "green"))(n = 299)
col_breaks = c(seq(-1,0,length=100),  # for red
  seq(0.01,0.8,length=100),           # for yellow
  seq(0.81,1,length=100))             # for green
sentence.list=sentence.list%>%filter(type=="inaug", File%in%sel.comparison, Term==1)
mat_data <- round(cor(sentence.list%>%filter(type=="inaug")%>%select(anger:trust)),2)
heatmap.2(mat_data, notecol="black",
          cellnote = mat_data,
          scale = "none", 
          col =my_palette , margin=c(6, 6), key=F,
          trace = "none", density.info = "none", breaks=col_breaks,
          notecex = 0.5)
par(mar=c(6, 6, 6, 4))

emo.means=colMeans(select(sentence.list%>%filter(type=="inaug", File=='HerbertHoover'), anger:trust)>0.01)
col.use=c("red2", "darkgoldenrod1", 
            "chartreuse3", "blueviolet",
            "darkgoldenrod2", "dodgerblue3", 
            "darkgoldenrod1", "darkgoldenrod1")
barplot(emo.means[order(emo.means)], las=2, col=col.use[order(emo.means)], horiz=T, main="Inaugural Speeches of Herbert Hoover", cex.main = 1.1)
par(mar=c(6, 6, 6, 4))

emo.means=colMeans(select(sentence.list%>%filter(type=="inaug", File=="FranklinDRoosevelt"), anger:trust)>0.01)
col.use=c("red2", "darkgoldenrod1", 
            "chartreuse3", "blueviolet",
            "darkgoldenrod2", "dodgerblue3", 
            "darkgoldenrod1", "darkgoldenrod1")
barplot(emo.means[order(emo.means)], las=2, col=col.use[order(emo.means)], horiz=T, main="Inaugural Speeches of Franklin D.Roosevelt", cex.main = 1)
par(mar=c(6, 6, 6, 4))

emo.means=colMeans(select(sentence.list%>%filter(type=="inaug", File=="GeorgeWBush"), anger:trust)>0.01)
col.use=c("red2", "darkgoldenrod1", 
            "chartreuse3", "blueviolet",
            "darkgoldenrod2", "dodgerblue3", 
            "darkgoldenrod1", "darkgoldenrod1")
barplot(emo.means[order(emo.means)], las=2, col=col.use[order(emo.means)], horiz=T, main="Inaugural Speeches of George W.Bush", cex.main = 1.1)
par(mar=c(6, 6, 6, 4))

emo.means=colMeans(select(sentence.list%>%filter(type=="inaug", File=='BarackObama'), anger:trust)>0.01)
col.use=c("red2", "darkgoldenrod1", 
            "chartreuse3", "blueviolet",
            "darkgoldenrod2", "dodgerblue3", 
            "darkgoldenrod1", "darkgoldenrod1")
barplot(emo.means[order(emo.means)], las=2, col=col.use[order(emo.means)], horiz=T, main="Inaugural Speeches of Barack Obama", cex.main = 1.1)

In the first heat map, Green stands for strong correlation while orange stands for weak correlation and the yellow is in the middle of green and orange. We assume correlation over 0.5 a relatively strong correlation. Among the inaugurations of the four presidents, fear and anger(0.57), disgust and anger(0.52), antipation and joy(0.7), joy and trust(0.58), antipation and trust(0.5) have strong bonds with each other.

Also in the four bar graphs which depict the emotions of the four presendents seperately, we can see that trust is the overwhelming No.1 emotion. In general, positive emotions are more than negative emotions. This accords with the fact the auguration is aim at enhancing morale and building confidence, at the mean time pointing the problems.

What’s interesting is that the emotion “fear” is very strong in Roosevelt’s and Obama’s inauguration. In Roosevelt’s speech, fear is even stonger than joy, in the third place. This can be explained that these two presidents were in the vortex of the economy collapse and fear is a prevailing emotion in that time.

We try to understand it in a real history context. When the stock market crashed in October 1929, President Herbert Hoover encouraged business leaders to take an interventionist approach to combat the impending economic emergency because “it is action that counts.” Over the next three years, however, Hoover worked unsuccessfully to mitigate the economic crisis of the Great Depression. Corporate welfare promises failed. State relief efforts dissipated. Not only was the federal government too small to handle the crisis, individuals and businesses across the political spectrum opposed federal intervention. Even then-governor of New York, Franklin Roosevelt, wrote privately, “I am very much opposed to the extension of Federal action in most economy social problems.” So when the Great Depression finally come, no wonder fear is the dominating feeling of the time.

Sentence length variation over the course of the speech, with emotions.

How the four presidents alternate between long and short sentences and how they shift between different sentiments in their speeches. It is interesting to note that some speeches are more colorful than others.

par(mfrow=c(4,1), mar=c(1,0,2,0), bty="n", xaxt="n", yaxt="n", font.main=1)
f.plotsent.len(In.list=sentence.list, InFile="HerbertHoover", 
               InType="inaug", InTerm=1, President="Herbert Hoover")
f.plotsent.len(In.list=sentence.list, InFile="FranklinDRoosevelt", 
               InType="inaug", InTerm=1, President="Franklin D Roosevelt")

f.plotsent.len(In.list=sentence.list, InFile="GeorgeWBush", 
               InType="inaug", InTerm=1, President="George W. Bush")
f.plotsent.len(In.list=sentence.list, InFile="BarackObama", 
               InType="inaug", InTerm=1, President="Barack Obama")

We can find from the graphs: 1. All the dominanting emotions are positive emotions, which is darkgoldenrod in the graph, even in the time fanancial situation is very dangerous. 2. Longer sentences usually convey positive emotions such as trust and anticipation. 3. All presidents know how to use long sentences and short sentences in turns to convey complicated emotions. The combination of long sentences and short sentences can make the speech more powerful.

What are the emotionally changed sentences?

print("HerbertHoover")

[1] "HerbertHoover"

speech.df=tbl_df(sentence.list)%>%
  filter(File=="HerbertHoover", type=="inaug", word.count>=4)%>%
  select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "The strong man must at all times be alert to the attack of insidious disease."                                                
[2] "Such progress, however, can continue only so long as business manifests its respect for law."                                 
[3] "THE FAILURE OF OUR SYSTEM OF CRIMINAL JUSTICEThe most malign of all these dangers today is disregard and disobedience of law."
[4] "It is an advance toward the highest conception of self-government."                                                           
[5] "the advancement of peace."                                                                                                    
[6] "If citizens do not like a law, their duty as honest men and women is to discourage its violation;"                            
[7] "It is an advance toward the highest conception of self-government."                                                           
[8] "the maintenance of ordered liberty;"                                                                                          

print("FranklinDRoosevelt")

[1] "FranklinDRoosevelt"

speech.df=tbl_df(sentence.list)%>%
  filter(File=="FranklinDRoosevelt", type=="inaug", Term==1, word.count>=5)%>%
  select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "Happiness lies not in the mere possession of money;"                 
[2] "Happiness lies not in the mere possession of money;"                 
[3] "Yet our distress comes from no failure of substance."                
[4] "These are the lines of attack."                                      
[5] "it lies in the joy of achievement, in the thrill of creative effort."
[6] "We are stricken by no plague of locusts."                            
[7] "Yet our distress comes from no failure of substance."                
[8] "In this dedication of a Nation we humbly ask the blessing of God."   

print("George W Bush")

[1] "George W Bush"

speech.df=tbl_df(sentence.list)%>%
  filter(File=="GeorgeWBush", type=="inaug", Term==1, word.count>=4)%>%
  select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "Abandonment and abuse are not acts of God;" "God bless you all, and God bless America." 
[3] "Where there is suffering, there is duty."   "Abandonment and abuse are not acts of God;"
[5] "God bless you all, and God bless America."  "Abandonment and abuse are not acts of God;"
[7] "Abandonment and abuse are not acts of God;" "God bless you all, and God bless America." 

print("Barack Obama")

[1] "Barack Obama"

speech.df=tbl_df(sentence.list)%>%
  filter(File=="BarackObama", type=="inaug", Term==1, word.count>=5)%>%
  select(sentences, anger:trust)
speech.df=as.data.frame(speech.df)
as.character(speech.df$sentences[apply(speech.df[,-1], 2, which.max)])

[1] "On this day, we gather because we have chosen hope over fear, unity of purpose over conflict and discord."
[2] "This is the journey we continue today."                                                                   
[3] "We remain the most prosperous, powerful nation on Earth."                                                 
[4] "Our Nation is at war against a far-reaching network of violence and hatred."                              
[5] "We remain the most prosperous, powerful nation on Earth."                                                 
[6] "Our Nation is at war against a far-reaching network of violence and hatred."                              
[7] "This is the meaning of our liberty and our creed;"                                                        
[8] "God bless you, and God bless the United States of America."                                               

Just as we have mentioned above, every time has its theme. This can be shown clearly in the emotionally changed sentences of each president. In Hoover’s time, disease and criminal is the primary enemy. While in Roosevelt’s time, words like “happiness”, “money”, “substance” reflect the society problem triggered by the weak economy. During Bush’s presidency, though most of the emtionally changed sentences are meaningless, the sentence “Where there is suffering, there is duty.” still reflects the fact of the society. Finally, in Obama’s speech, the sentences usually aim at reaching a consensus and building confidence.

sentence.list

Step 5: Data analysis — Topic modeling

For topic modeling, we try to find the common part in the different inaugurations.

corpus.list=sentence.list[2:(nrow(sentence.list)-1), ]
sentence.pre=sentence.list$sentences[1:(nrow(sentence.list)-2)]
sentence.post=sentence.list$sentences[3:(nrow(sentence.list)-1)]
corpus.list$snipets=paste(sentence.pre, corpus.list$sentences, sentence.post, sep=" ")
rm.rows=(1:nrow(corpus.list))[corpus.list$sent.id==1]
rm.rows=c(rm.rows, rm.rows-1)
corpus.list=corpus.list[-rm.rows, ]

Text mining

docs <- Corpus(VectorSource(corpus.list$snipets))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

For the trust reposed in me I will return the courage and the devotion that befit the time. I can do no less. We face the arduous days that lie before us in the warm courage of national unity;

Text basic processing

#remove potentially problematic symbols
docs <-tm_map(docs,content_transformer(tolower))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

it has not been the path for the fainthearted, for those who prefer leisure over work or seek only the pleasures of riches and fame. rather, it has been the risk-takers, the doers, the makers of things--some celebrated, but more often men and women obscure in their labor--who have carried us up the long, rugged path toward prosperity and freedom. for us, they packed up their few worldly possessions and traveled across oceans in search of a new life.

#remove punctuation
docs <- tm_map(docs, removePunctuation)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

we honor them not only because they are guardians of our liberty but because they embody the spirit of service a willingness to find meaning in something greater than themselves and yet at this moment a moment that will define a generation it is precisely this spirit that must inhabit us all for as much as government can do and must do it is ultimately the faith and determination of the american people upon which this nation relies

#Strip digits
docs <- tm_map(docs, removeNumbers)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

our goods and services no less needed than they were last week or last month or last year our capacity remains undiminished but our time of standing pat of protecting narrow interests and putting off unpleasant decisions that time has surely passed

#remove stopwords
docs <- tm_map(docs, removeWords, stopwords("english"))
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

america   best matches  commitment  principle   concern  civility  civil society demands    us good will  respect fair dealing  forgiveness  seem  believe   politics can afford   petty    time  peace  stakes   debates appear small

#remove whitespace
docs <- tm_map(docs, stripWhitespace)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

 abiding faith capacity integrity high purpose fears future country bright hope

#Stem document
docs <- tm_map(docs,stemDocument)
writeLines(as.character(docs[[sample(1:nrow(corpus.list), 1)]]))

encourag respons search scapegoat call conscienc though requir sacrific bring deeper fulfil

Topic modeling

Gengerate document-term matrices.

library(tm)
library(NLP)
library(magrittr)
dtm <- DocumentTermMatrix(docs)
#convert rownames to filenames#convert rownames to filenames
rownames(dtm) <- paste(corpus.list$type, corpus.list$File,
                       corpus.list$Term, corpus.list$sent.id, sep="_")
rowTotals <- apply(dtm , 1, sum) #Find the sum of words in each Document
dtm  <- dtm[rowTotals> 0, ]
corpus.list=corpus.list[rowTotals>0, ]

Run LDA

#Set parameters for Gibbs sampling
burnin <- 4000
iter <- 2000
thin <- 500
seed <-list(2003,5,63,100001,765)
nstart <- 5
best <- TRUE
#Number of topics
k <- 15
#Run LDA using Gibbs sampling
ldaOut <-LDA(dtm, k, method="Gibbs", control=list(nstart=nstart, 
                                                 seed = seed, best=best,
                                                 burnin = burnin, iter = iter, 
                                                 thin=thin))
#write out results
#docs to topics
ldaOut.topics <- as.matrix(topics(ldaOut))
table(c(1:k, ldaOut.topics))

 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 
48 46 39 37 50 41 33 27 24 24 29 33 38 24 35 

write.csv(ldaOut.topics,file=paste("/Users/wangzhishan/Desktop/Spr2017-Proj1-iriswang0501/out/LDAGibbs",k,"DocsToTopics.csv"))
#top 6 terms in each topic
ldaOut.terms <- as.matrix(terms(ldaOut,20))
write.csv(ldaOut.terms,file=paste("/Users/wangzhishan/Desktop/Spr2017-Proj1-iriswang0501/out/LDAGibbs",k,"TopicsToTerms.csv"))
#probabilities associated with each topic assignment
topicProbabilities <- as.data.frame(ldaOut@gamma)
write.csv(topicProbabilities,file=paste("../out/LDAGibbs",k,"TopicProbabilities.csv"))

cannot open file '../out/LDAGibbs 15 TopicProbabilities.csv': No such file or directoryError in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection

terms.beta=ldaOut@beta
terms.beta=scale(terms.beta)
topics.terms=NULL
for(i in 1:k){
  topics.terms=rbind(topics.terms, ldaOut@terms[order(terms.beta[i,], decreasing = TRUE)[1:7]])
}
topics.terms

      [,1]         [,2]      [,3]        [,4]          [,5]      
 [1,] "educ"       "health"  "stabil"    "adapt"       "competit"
 [2,] "oath"       "hour"    "affirm"    "inevit"      "someth"  
 [3,] "recoveri"   "field"   "immedi"    "interdepend" "temper"  
 [4,] "come"       "carri"   "futur"     "gift"        "word"    
 [5,] "less"       "year"    "birth"     "man"         "last"    
 [6,] "confront"   "creat"   "instead"   "reward"      "show"    
 [7,] "law"        "citizen" "liquor"    "due"         "evil"    
 [8,] "measur"     "meet"    "action"    "congress"    "execut"  
 [9,] "activ"      "amend"   "essenti"   "consid"      "indic"   
[10,] "order"      "sound"   "creation"  "major"       "path"    
[11,] "leadership" "anoth"   "complex"   "singl"       "applic"  
[12,] "achiev"     "big"     "honest"    "joy"         "materi"  
[13,] "find"       "commit"  "love"      "degre"       "discharg"
[14,] "definit"    "grow"    "resourc"   "convict"     "farm"    
[15,] "peac"       "prosper" "contribut" "cultur"      "burden"  
      [,6]       [,7]       
 [1,] "lessen"   "movement" 
 [2,] "accumul"  "afraid"   
 [3,] "threat"   "animos"   
 [4,] "set"      "account"  
 [5,] "whose"    "capit"    
 [6,] "soil"     "togeth"   
 [7,] "creed"    "disregard"
 [8,] "clear"    "fulfil"   
 [9,] "influenc" "thereof"  
[10,] "reserv"   "statesmen"
[11,] "assert"   "boy"      
[12,] "minist"   "perform"  
[13,] "arrog"    "decenc"   
[14,] "fit"      "natur"    
[15,] "fulli"    "mutual"   

ldaOut.terms

      Topic 1   Topic 2    Topic 3      Topic 4   Topic 5   
 [1,] "govern"  "spirit"   "nation"     "time"    "america" 
 [2,] "public"  "faith"    "world"      "hope"    "common"  
 [3,] "busi"    "today"    "advanc"     "come"    "everi"   
 [4,] "home"    "work"     "peopl"      "stori"   "less"    
 [5,] "cooper"  "yet"      "greater"    "generat" "year"    
 [6,] "protect" "servic"   "respect"    "promis"  "face"    
 [7,] "right"   "much"     "way"        "end"     "can"     
 [8,] "econom"  "oath"     "effort"     "long"    "let"     
 [9,] "educ"    "presid"   "make"       "carri"   "now"     
[10,] "health"  "thank"    "polici"     "day"     "children"
[11,] "power"   "moment"   "realiz"     "endur"   "danger"  
[12,] "continu" "american" "recoveri"   "futur"   "birth"   
[13,] "equal"   "begin"    "field"      "great"   "man"     
[14,] "servic"  "hour"     "becom"      "pass"    "father"  
[15,] "state"   "make"     "neighbor"   "freedom" "last"    
[16,] "regul"   "storm"    "move"       "gift"    "lead"    
[17,] "stabil"  "see"      "put"        "word"    "women"   
[18,] "develop" "affirm"   "understand" "courag"  "crisi"   
[19,] "group"   "humbl"    "dedic"      "fear"    "travel"  
[20,] "promot"  "old"      "oblig"      "forward" "generat" 
      Topic 6    Topic 7   Topic 8     Topic 9     Topic 10     
 [1,] "will"     "law"     "shall"     "system"    "upon"       
 [2,] "new"      "citizen" "may"       "justic"    "american"   
 [3,] "know"     "respons" "nation"    "feder"     "purpos"     
 [4,] "build"    "problem" "one"       "part"      "instrument" 
 [5,] "power"    "failur"  "measur"    "enforc"    "first"      
 [6,] "courag"   "support" "meet"      "whole"     "relat"      
 [7,] "economi"  "enforc"  "duti"      "effect"    "order"      
 [8,] "remain"   "crimin"  "need"      "caus"      "parti"      
 [9,] "histori"  "larg"    "action"    "mani"      "propos"     
[10,] "choic"    "duti"    "author"    "organ"     "settlement" 
[11,] "seek"     "share"   "constitut" "activ"     "establish"  
[12,] "pledg"    "elect"   "congress"  "amend"     "controversi"
[13,] "demand"   "state"   "seek"      "essenti"   "justic"     
[14,] "america"  "accept"  "ask"       "local"     "question"   
[15,] "road"     "liquor"  "execut"    "may"       "special"    
[16,] "without"  "extend"  "given"     "nation"    "chang"      
[17,] "confront" "men"     "emerg"     "administr" "practic"    
[18,] "creat"    "life"    "balanc"    "state"     "among"      
[19,] "side"     "due"     "bring"     "abus"      "court"      
[20,] "age"      "evil"    "clear"     "delay"     "intern"     
      Topic 11     Topic 12   Topic 13    Topic 14     Topic 15   
 [1,] "must"       "fail"     "civil"     "can"        "peac"     
 [2,] "peopl"      "place"    "find"      "govern"     "progress" 
 [3,] "ideal"      "lie"      "direct"    "small"      "world"    
 [4,] "leadership" "trust"    "call"      "help"       "prosper"  
 [5,] "good"       "true"     "countri"   "use"        "human"    
 [6,] "mani"       "valu"     "live"      "must"       "countri"  
 [7,] "everi"      "offic"    "life"      "task"       "unit"     
 [8,] "increas"    "polit"    "freedom"   "war"        "old"      
 [9,] "opportun"   "restor"   "concern"   "accomplish" "new"      
[10,] "belief"     "standard" "commit"    "work"       "toward"   
[11,] "confid"     "money"    "great"     "land"       "contribut"
[12,] "aspir"      "high"     "communiti" "whether"    "limit"    
[13,] "disciplin"  "moral"    "god"       "beyond"     "democraci"
[14,] "give"       "way"      "knowledg"  "hand"       "return"   
[15,] "secur"      "fals"     "import"    "stimul"     "cultur"   
[16,] "believ"     "hand"     "thing"     "best"       "desir"    
[17,] "follow"     "profit"   "interest"  "face"       "die"      
[18,] "constant"   "scienc"   "present"   "definit"    "earth"    
[19,] "uniti"      "men"      "express"   "grow"       "sure"     
[20,] "class"      "mere"     "reach"     "liberti"    "advanc"   

Based on the most popular terms and the most salient terms for each topic, we assign a hashtag to each topic. This part require manual setup as the topics are likely to change.

topics.hash=c("Economy", "America", "Defense", "Belief", "Election", "Patriotism", "Unity", "Government", "Reform", "Temporal", "WorkingFamilies", "Freedom", "Equality", "Misc", "Legislation")
corpus.list$ldatopic=as.vector(ldaOut.topics)
corpus.list$ldahash=topics.hash[ldaOut.topics]
colnames(topicProbabilities)=topics.hash
corpus.list.df=cbind(corpus.list, topicProbabilities)

Clustering of topics

par(mar=c(1,2,1,1))
topic.summary=tbl_df(corpus.list.df)%>%
              filter(type%in%c("nomin", "inaug"), File%in%sel.comparison)%>%
              select(File, Economy:Legislation)%>%
              group_by(File)%>%
              summarise_each(funs(mean))
topic.summary=as.data.frame(topic.summary)
rownames(topic.summary)=topic.summary[,1]
# [1] "Economy"         "America"         "Defense"         "Belief"         
# [5] "Election"        "Patriotism"      "Unity"           "Government"     
# [9] "Reform"          "Temporal"        "WorkingFamilies" "Freedom"        
# [13] "Equality"        "Misc"            "Legislation"       
topic.plot=c(1, 13, 9, 11, 8, 3, 7)
print(topics.hash[topic.plot])

[1] "Economy"         "Equality"        "Reform"         
[4] "WorkingFamilies" "Government"      "Defense"        
[7] "Unity"          

heatmap.2(as.matrix(topic.summary[,topic.plot+1]), 
          scale = "column", key=F, 
          col = bluered(100),
          cexRow = 0.9, cexCol = 0.9, margins = c(8, 8),
          trace = "none", density.info = "none")

In the heatmap, red indicates higher correlation and blue indicates lower correlation (possibly negative values). It is obvious that President Hoover likes discussing Unity, Reform and Economy. President Roosevelt paid more attention to Government and Defenses. President Bush liked talking about Equality. President Obama did not have a preference towards the topics but he seemed to talk less about Defences compared to other presidents. Maybe it is because he is a president of peace time.

# [1] "Economy"         "America"         "Defense"         "Belief"         
# [5] "Election"        "Patriotism"      "Unity"           "Government"     
# [9] "Reform"          "Temporal"        "WorkingFamilies" "Freedom"        
# [13] "Equality"        "Misc"            "Legislation"       
 
par(mfrow=c(5, 1), mar=c(1,1,2,0), bty="n", xaxt="n", yaxt="n")
library(hash)
topic.plot=c(1, 13, 14, 15, 8, 9, 12)
print(topics.hash[topic.plot])

[1] "Economy"     "Equality"    "Misc"        "Legislation"
[5] "Government"  "Reform"      "Freedom"    

speech.df=tbl_df(corpus.list.df)%>%filter(File=="HerbertHoover", type=="inaug",Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1], 
             xlab="Sentences", ylab="Topic share", main="HerbertHoover")

[1] 0.06942528 0.13885055 0.20827583 0.27770111 0.34712638 0.41655166
[7] 0.48597694

speech.df=tbl_df(corpus.list.df)%>%filter(File=="FranklinDRoosevelt", type=="inaug", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
            xlab="Sentences", ylab="Topic share", main="FranklinDRoosevelt")

[1] 0.06173981 0.12347963 0.18521944 0.24695926 0.30869907 0.37043889
[7] 0.43217870

speech.df=tbl_df(corpus.list.df)%>%filter(File=="GeorgeWBush", type=="inaug", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1], 
            xlab="Sentences", ylab="Topic share", main="George W Bush")

[1] 0.051246 0.102492 0.153738 0.204984 0.256230 0.307476 0.358722

speech.df=tbl_df(corpus.list.df)%>%filter(File=="BarackObama", type=="inaug", Term==1)%>%select(sent.id, Economy:Legislation)
speech.df=as.matrix(speech.df)
speech.df[,-1]=replace(speech.df[,-1], speech.df[,-1]<1/15, 0.001)
speech.df[,-1]=f.smooth.topic(x=speech.df[,1], y=speech.df[,-1])
plot.stacked(speech.df[,1], speech.df[,topic.plot+1],
            xlab="Sentences", ylab="Topic share", main="Barack Obama")

[1] 0.05795373 0.11590745 0.17386118 0.23181491 0.28976863 0.34772236
[7] 0.40567609

We can have a very direct impression about which president focused on which topic more clearly. The key word for President Hoover is “economy”. President Roosevelt paid more attention to “freedom” at the early stage of the speech, loved to talk more about “Misc” in the middle of the speech, and discussed “Government” at the end of the speech. President Bush liked talking about “Equality” all the way. President Obama tended to talk “Misc” more.

speech.df=tbl_df(corpus.list.df)%>%filter(type=="nomin", word.count<20)%>%select(sentences, Economy:Legislation)
as.character(speech.df$sentences[apply(as.data.frame(speech.df[,-1]), 2, which.max)])

character(0)

names(speech.df)[-1]

 [1] "Economy"         "America"         "Defense"        
 [4] "Belief"          "Election"        "Patriotism"     
 [7] "Unity"           "Government"      "Reform"         
[10] "Temporal"        "WorkingFamilies" "Freedom"        
[13] "Equality"        "Misc"            "Legislation"    

library(factoextra)
library(ggplot2)
presid.summary=tbl_df(corpus.list.df)%>%
  filter(type=="inaug", File%in%sel.comparison)%>%
  select(File, Economy:Legislation)%>%
  group_by(File)%>%
  summarise_each(funs(mean))
presid.summary=as.data.frame(presid.summary)
rownames(presid.summary)=as.character((presid.summary[,1]))
km.res=kmeans(scale(presid.summary[,-1]), iter.max=200,
              2)
fviz_cluster(km.res, 
             stand=T, repel= TRUE,
             data = presid.summary[,-1],
             show.clust.cent=FALSE)

It seems time has a very compelling influence on the inauguration speeches of the presidents. People in the same decades tend to talk about similiar topics. Time change and the culture change, the issues change. Every time has its unique emotion.

Step 6: Wordcloud

folder.path="/Users/wangzhishan/Desktop/applied data science"
speeches=list.files(path = folder.path, pattern = "*.txt")
prex.out=substr(speeches, 6, nchar(speeches)-4)
ff.all<-Corpus(DirSource(folder.path))

wordcloud

folder.path1="/Users/wangzhishan/Desktop/Spr2017-Proj1-iriswang0501/data/inaugs"
folder.path2="/Users/wangzhishan/Desktop/Spr2017-Proj1-iriswang0501/data/inaugurals"
speeches1=list.files(path = folder.path1, pattern = "*.txt")
prex.out1=substr(speeches1, 6, nchar(speeches1)-4)
ff.all1<-Corpus(DirSource(folder.path1))
speeches2=list.files(path = folder.path2, pattern = "*.txt")
prex.out2=substr(speeches2, 6, nchar(speeches2)-4)
ff.all2<-Corpus(DirSource(folder.path2))

ff.all1<-tm_map(ff.all1, stripWhitespace)
ff.all1<-tm_map(ff.all1, content_transformer(tolower))
ff.all1<-tm_map(ff.all1, PlainTextDocument)
ff.all1<-tm_map(ff.all1, removeWords, stopwords("english"))
ff.all1<-tm_map(ff.all1, removeWords, character(0))
ff.all1<-tm_map(ff.all1, removePunctuation)
ff.all2<-tm_map(ff.all2, stripWhitespace)
ff.all2<-tm_map(ff.all2, content_transformer(tolower))
ff.all2<-tm_map(ff.all2, PlainTextDocument)
ff.all2<-tm_map(ff.all2, removeWords, stopwords("english"))
ff.all2<-tm_map(ff.all2, removeWords, character(0))
ff.all2<-tm_map(ff.all2, removePunctuation)
tdm.all1<-TermDocumentMatrix(ff.all1)
tdm.all2<-TermDocumentMatrix(ff.all2)
tdm.tidy1=tidy(tdm.all1)
tdm.tidy2=tidy(tdm.all2)
tdm.overall1=summarise(group_by(tdm.tidy1, term), sum(count))
tdm.overall2=summarise(group_by(tdm.tidy2, term), sum(count))

wordcloud(tdm.overall1$term, tdm.overall1$`sum(count)`,
          scale=c(5,0.5),
          max.words=100,
          min.freq=1,
          random.order=FALSE,
          rot.per=0,
          use.r.layout=T,
          random.color=FALSE,
          colors=brewer.pal(9,"Greens"))

wordcloud(tdm.overall2$term, tdm.overall2$`sum(count)`,
          scale=c(5,0.5),
          max.words=100,
          min.freq=1,
          random.order=FALSE,
          rot.per=0,
          use.r.layout=T,
          random.color=FALSE,
          colors=brewer.pal(9,"Greens"))

It seems the word cloud of the four presidents during the financial problems is highly similiar to the rest of the presidents. This shows the common of the inaugurals of the United States. The inauguration speech usually marks the beginning of the president term of office. Without exception, the new president all choose to consolidate confidence and strengthen cooperation, to embark a new great journey.

Step 7: Summary

The function of inaugural addresses is precisely to express those cultural features of American politics, political scientists and others have long noted, such transient features of American politics as the incoming president’s policy agenda. The inauguration speeched of four presidents have some in common with the others, while also have some unique parts. First, we try to analyze the speeches from the length of the sentences. And in short sentences, we get a glimpse of the time. Next, from sentiment analysis, we find the unique emotions during the economic collapse and the common emotions of all time. In the topic modeling, the topics link presidential inaugural addresses together as one tradition. We do clustering of topics and find the topic tend to be influenced by the time a lot. Finally we perform a wordcloud to get a direct comparision of the speeches we selected and the whole speeches. In general, they are pretty similiar. The more important factor is that certain features of American politics seem so permanent and pervasive.